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As our society grapples with rapid technological 
change, assessing technology literacy has emerged as a 
K-12 education issue. The No Child Left Behind Act 
specifies as a goal that every child will be technologically 
literate by the end of the eighth grade, business pleads for 
a workforce that can use information and communica- 
tions technology (ICT), and the education community 
itself recognizes the importance of establishing a founda- 
tion for lifelong learning. 

The Institute for the Advancement of Emerging 
Technologies in Education (lAETE), housed at 
Edvantia’s Appalachia Educational Laboratory, addressed 
the need to assess technology literacy in an e-mail-based 
panel held during one week in September 2005. (Further 
exploration of this topic can be found in companion 
articles in lAETE’s online publication, InSight, at 
www.edvantia.org/insight/). 

The conversation began with the importance of 
defining technology literacy, identifying a range of 
definitions rather than precise limits. There was, how- 
ever, consensus that ICT literacy— which can be viewed 
as one component of technological literacy— implies both 
basic computer skills and the cognitive abilities associ- 
ated with using those tools for learning and communica- 
tion. The panelists then turned to the emerging possibili- 
ties for assessing via simulation, the new design processes 
needed for any computer-based assessment, and the lack 
of research to assist assessment developers in this new 
territory. Mary Axelson moderated the discussion among 
the following panel members: 

• Martin Ripley heads the eStrategy Unit of the United 
Kingdom’s Quality and Curriculum Authority (QCA). In 
that capacity, he has led groundbreaking work on assessing 
technology skills and higher order thinking via simulations. 
His work differs from other such work because it looks at 



large patterns of activity rather than solely at a final answer. 
See www.qca.org.uk and www.ks3ictpilot.com. 

• Greg Pearson, program officer with the National Academy 
of Engineering in Washington, DC, serves as the respon- 
sible staff officer for Assessing Technological Literacy in the 
United States, a study funded by the National Science 
Foundation (NSF), and the State Educators’ Symposium on 
Technological Literacy project, which is funded by the U.S. 
Department of Education. He is the lead author of Techni- 
cally Speaking: Why All Americans Need to Know More About 
Technology (National Academy Press, 2002) and a related 
report on assessment to be published in 2005. See 
www.nae.edu/techlit. 

• Margaret Honey is vice president of the Education 
Development Center (EDC) and director of EDC’s Center 
for Children and Technology (www.edc.org/CCT). In the 
latter capacity, she is also the coauthor of Assessment of 21st 
Century Skills: The Current Landscape, a report of assess- 
ments published by the Partnership for 21st Century Skills 
(www.21stcenturyskills.org). 

• Kate Kemker, director of instructional technology for the 
Florida Department of Education, has worked with a state 
team that created a simulation-based assessment of technol- 
ogy skills for teachers, known as the Inventory of Teacher 
Technology Skills. She now works on a similar inventory 
for eighth-grade students to be used in the spring of 2006. 
See www.flstar.org. 

Defining Technoiogy 

Technically Speaking, writes Pearson, “spends many 
pages defining and defending an expansive notion of 
technology and technological literacy.” Elsewhere, he 
summarizes technology literacy “as a process of modify- 
ing the natural world to satisfy human needs and wants, 
and of course the products that result from that process.” 
For the National Academies, technology can be a 
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computer, or it can be a shovel, a pencil, paper, or the 
process of identifying chromosomes on a strand of 
DNA. Writes Pearson, “As this panel illustrates in 
microcosm, when the word technology is mentioned in 
the context of U.S. education, it is associated almost 
exclusively with computers and IT (information technol- 
ogy) or ICT literacy.This is not wrong . . . but it is a 
rather narrow view of things.” From his work on 
assessment, Pearson notes that an emerging priority is a 
familiarity with the “technological design process, which 
has many parallels with science inquiry.” 

Pearson also points out that, beyond this panel, the 
National Academies’ perspective has supporters. 

In the interest of complete honesty, the Academies 
are far from the firstto describe and promote this 
larger view of technology .Philosophers of 
technology such as Langdon Winner, historians of 
technology like Thomas Hughes, and technology 
critics like Neil Postman have all taken this larger 
view. So has the STS (science, technology, and 
society) movement in U.S. postsecondary educa- 
tion, and the International Technology Education 
Association, which represents technology educa- 
tion teachers in this country. 

No matter the definition, people working in a K-12 
environment make ICT their priority. Writes Honey, 

In contrast to Greg, I’m choosing to highlight a set 
of fairly specific skills and competencies that 
students need. I don’t by any means disagree with 
Greg— I like the breadth of his argument and ideas. 

I’m motivated to write more specifically only 
because I see too much of what is NOT being 
taught in schools. In too many schools we are 
stuck with a very 1980s notion of technology 
competency— a kind of basic facts approach to 
technology — that’s not serving our kids well. 

Kemker, too, sees a pressing need to establish a 
foundation of computer skills. In Florida, for example, 
the state hopes to put its high-stakes test online, and an 
inventory of student computer skills could help ensure 
that the computer-based format does not hinder student 
performance. However, she’s also wary of the momen- 
tum stopping there: 

The discussion from my perspective is, do you 
measure all of these skills at once with a perfor- 
mance-based solution within scenario-based 
settings? Or do you begin with basic computer 
skills, then communication, basically in a hierar- 
chal method? The situation that we face in K-12 
education is that if we begin with one area then 
precedence has been established in which those 
skills tend to have greater value than the other 
skills. 



Technology Literacy or Technological Literacy? 

Panelists were happy with either term. This 
document, however, will use technological literacy to 
reference the National Academies’ inclusive view of 
technology and technology literacy for the subset 
related to information and communications technolo- 
gies. Kemker lobbied for “digital literacy” for the 
latter. 



Defining Literacy as Higher Order Thinking 

Though there was not wide agreement on the scope 
of the word technology, all panelists did share an under- 
standing of literacy as something significantly beyond 
basic skills. Ripley has a particular interest in the higher 
order thinking that high-tech tools make possible. 
Observing that schools now focus more on developing 
student computer skills to support learning across 
curricula rather than studying about computers, Ripley 
asks. 

So, can we usefully ponder what subject will 
replace IT, and why?My view is that the subject 
must change its outlook from training students 
predominantly in the skills and capabilities that 
arise from the existence of personal computers. 

Instead we should look for the subject IT to 
concern itself increasingly with the growing range 
of technologies (mobile devices, blogs, video). It 
should concern itself with the uses to which 
technology is put (work, leisure, recreation, 
purchase). And it should concern itself with the 
facilities (or capabilities) that those technologies 
provide to students and adults (voice, visual 
communication, decision making, choice, responsi- 
bility). 

After quoting a job description for an engineer at Boeing 
(and safely assuming education prepares people for jobs), 
Ripley summarizes, “In other words, we must expect 
technology to help our students to be ‘flexible’ and to be 
‘curious.’” 

Pearson’s reports identify “three interdependent 
dimensions” of technological literacy: knowledge, 
capability, and ways of thinking and acting. Pearson 
notes that the capability dimension relates to abilities 
that may be easier to assess. According to Pearson, the 
committee noted that a “technologically literate person 
would 

• have hands-on skills, such as using a computer for word 
processing and surfing the Internet and be able to operate a 
variety of home and office appliances, 
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• be able to identify and fix simple mechanical or technologi- 
cal problems at home or work, and 

• be able to apply basic mathematical concepts related to 
probability, scale, and estimation to make informed 
judgments about technological risks and benefits.” 

Honey observes that students today have almost 
universal access to unfiltered and unsubstantiated 
information. No longer do they receive their informa- 
tion solely from teachers or librarians, or through 
textbooks or print-based reference materials. She contin- 
ues, 

To navigate through this wealth (or glut) of data, 
draw conclusions, and communicate with others 
proficiently, teachers and students require whole new 
sets of skills surrounding accessing, interpreting, 
analyzing and evaluating complex sets of images, 
words, numbers and sounds in meaningful ways. 

She identified some of these skills as communicating 
effectively, analyzing and interpreting data, understand- 
ing models and simulations (computational literacy), 
managing and prioritizing tasks, engaging in problem 
solving, and ensuring security and safety. Her work with 
the Partnership for 21st Century Skills identifies ICT 
learning skills as related to thinking and problem-solving 
skills, information and communication skills, and 
interpersonal and self-direction skills. Honey and the 
Partnership emphasize that these skills are a priority and 
can be addressed through core subjects as well as in areas 
they refer to as 21st Century Content: global awareness; 
financial, economic, and business literacy; and civic 
literacy. ICT learning skills are interwoven with technol- 
ogy because their pursuit is so often assisted by high-tech 
tools. 

The Promise of Simulation-Based 
Assessment 

An assessment environment that simulates common 
software applications is an ideal and obvious way to 
investigate a student’s abilities with computer applica- 
tions. Assessments that utilize only one brand or model 
of software would be severely limited, as there is no 
standard for any common software applications, such as 
word processors, spreadsheets, or databases. A simulation 
also allows greater leverage for the capturing and report- 
ing of multiple points of data, functions that would be 
complex or expensive to retrofit to existing software. 
After developing such projects, Kemker and Ripley are 
eager to apply the environment to other school subjects. 
Kemker sees a fit with music. Ripley wants students to 
explore performances of Shakespeare. Both also see 
potential for science. More broadly speaking, however. 



simulations may hold promise for assessing complex 
cognition. 

Ripley’s assessments take place in the virtual world 
of Pepford, where work assignments arrive via e-mail 
and a “walled garden” provides a virtual world of Web 
sites and applications. Writes Ripley, 

The use of simulations is a key development. A 
simulator provides the context within which 
authentic assessment tasks can be designed and 
delivered to students. It alsofacilitates the develop- 
ment of assessment tasks that invite students to 
combine a range of capabilities and skills. The 
combination of these two aspects enables us to 
assess higher order IT capabilities, such as choice 
or communication. 

The test records and scores the actions that the 
student takes while completing the test. For 
example, a higher order capability for a 14-year-old 
student in England includes designing a system for 
someone else to use. That capability in turn 
includes an assessment of the end-user’s require- 
ments. This we assess dynamically in the virtual 
world of Pepford by collecting evidence of the 
student researching into those requirements, by 
sending and receiving emails to ask about require- 
ments, by the student refining the system to meet 
requirements and so on. To achieve this we have 
worked with an extensive range of teachers to 
document the processes that students go through 
when producing eloquent or satisfactory responses 
to the task set. We use this [data from teachers] to 
create a matrix of plausible routes that a student 
will take when en route to complete a satisfactory 
(or better) response. 

Simulation, of course, is not the only possible 
method to assess technology literacy. As Pearson points 
out, “I first will disagree slightly with Mary’s contention 
that assessment of higher-order thinking requires ‘new’ 
assessments. Assessments that get at the more complex 
aspects of student thinking already exist — in instruments 
that creatively use extended and open-ended response 
items, and in some portfolio techniques.” He points to 
an assessment of design capability by the International 
Baccalaureate as an example. And Pearson observes that 
while portfolio assessments are often viewed as limited in 
terms of providing valid and reliable data in a high-stakes 
arena, they could face fewer problems “if the rubrics for 
evaluating them are carefully thought through and 
teachers/evaluators are trained on the rubrics’ use.” 
Honey’s report. The Assessment of 21st Century Skills: The 
Current Landscape (www.21stcenturyskills.org/ assess21), 
also identifies an array of assessments for the 21st 
Century Skills other than ICT literacy. 
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Working with a New Design Process 

should one wish to pursue simulation or other 
computer-based applications, such as virtual environ- 
ments or multiplayer, role-playing applications, both 
Ripley and Kemker forewarn that the entire design 
process changes. Writes Ripley, 

In every e-test development project I have worked 
on it is possible to get all of the senior staff 
(designers, technologists, psychometricians, 
trainers, teachers) together regularly. These core 
teams number around 15-20 individuals and can be 
ledeffectively as a single team. There is no parallel 
in my experience with paper-based tests, which are 
often developed in a linear process, with different 
teams (involving large numbers of people) respon- 
sible for the various stages of development, 
administration and marking. 

Kemker, too, describes such teams. Echoing advice 
given at an lAETE symposium' on formative, online 
assessment, she focuses on the importance of the project 
manager. 

The development of online assessments requires a 
very diverse team of individuals. It is imperative 
that each member of the team be an expert in 
specific fields, such as measurement, technology, 
and curriculum. Last year, as we developed the 
online assessment tool for teacher technology 
skills, the team consisted of these individuals in 
addition to teachers. 

However, the key ingredient was to have a leader 
that focused on the scope of the project, or 
basically the project manager. This individual 
understood every aspect of the process, so that the 
focus remained the development of a performance 
based tool . . . 

It is critical to have the team leader be a measure- 
ment expert so that the team does not stray from 
the original scope of the project. In addition the 
team needs to be forward thinking so that the tool 
developed will have longevity. 

In Kemker’s experience, bringing technologists and 
assessment experts together sparked a great deal of 
creativity and discussions of new possibilities for assess- 
ment. Ripley, however, lamented the lack of creativity 
from technologists in his experience. “The concepts, the 
ideas, the vision have come predominantlyfrom the 
assessment experts and teachers. The quest for technology 
solutions to the many complex measurement problems 
and barriers has come from the assessment experts.” 



'Axelson, M. (2005). Online standards-based formative 
assessment conference proceedings. Charleston, WV: 
Edvantia. 



Ripley also cautions that change must go beyond the 
design group. The types of tests his work group designs 
pose problems beyond the “one correct answer” para- 
digm that dominates assessments for accountability 
purposes. He writes. 

Assessments which use simulations are a radical de- 
velopment. Delivering high-stakes assessments to all 
schools onscreen is a departure from current practice. 

This type of test development project involves sys- 
temic, nationwideinnovation and change. The na- 
ture and extent of innovation inthis type of 
redesignchallengespsychometric expertise and wis- 
dom; it requires teachers to adapt their teaching, 
andit requirespupils to reconsiderideas of whattests 
involve; it requires a national school hardware infra- 
structure that is as robust as someinternational bank- 
ing systems; it requiresnetwork managers and techni- 
cal staff in schools who can support the system. So, 
this is what I mean by‘ systemic innovation.’ 

Another way of describing such systemic innova- 
tion is to say that it requires a clear acknowledge- 
ment, including from national political and educa- 
tional sponsors, that in the beginning there are no 
guarantees of success and there is absolutely no 
track record of successful delivery. From the be- 
ginning, there needs to be flexibility within the 
proposed approach toreconsider, review and learn 
lessons. A willingness to admit mistakes and start 
afresh when needed. And there needs to be enough 
time in theearly development phases and initial 
roll-out, to prove the conceptual approach and to 
build the track record. 

We started our project to develop an ICT test for 
13-year-olds in 2001. We aim to have it rolled-out 
toall of our secondary schools (high schools) by 
2008. 1 do not believe that we could achieve our 
aims in less than seven years! 

Establishing a Research Base 

Ripley’s comment above, “The nature and extent of 
innovation in this type of redesign challenges psychomet- 
ric expertise and wisdom,” is a reference to the lack of 
research on the validity or reliability of assessment via 
simulation. Pointing out the increasing presence of simu- 
lation for instruction, Pearson observes that “the use of 
simulation in educational assessment probably requires 
quite a bit more study before it can be used with confi- 
dence. The assessment literature is mostly silent on simu- 
lation and essential psychometric issues such as reliabili- 
ty, validity, and precision.” Ripley suggests the need for 
research on all computer-based testing: 

Greg has elsewhere madea good point about the 
general lack of psychometric research and empiri- 
cally based evidence to support the most adventur- 
ous forms of e-test development. I see that defi- 
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ciency in almost all aspects of e-test development. 

.. . In England I like to remind colleagues that 
paper-based testing has been in use for 140 years. 

The science and business of paper-based testing is 
well refined andwell evidenced. A person experi- 
enced in managing, researching or administering 
one set of paper-based tests is likely to be able to 
move in a straightforward way to a similar role in 
relation to other paper-based tests. In contrast, 
there is littlestability yet in e-testing. The business 
processes are immature. Even when the e-test 
consists of closed-response or multi-choiceitems, 
the on-screen delivery involves making design 
decisions on the basis of emergent thinking, not on 
industry best practice standards (which do not yet 
exist). 

Ripley advises that research focus on “sources of 
difficulty” (the full range of demands carried by a test, 
but not part of the targeted assessment domain), the 
validity and reliability of measurement methods used in 
simulation-based assessments, and the possibilities of 
expanding it to domains beyond ICT. 

Pearson offered one illustration of the numerous 
research questions that need to be considered for simula- 
tion-based assessment: “Can each action taken by an 
individual in a simulation or game be treated as a test 
item and its correctness be judged by an on-demand, real- 
time assessment of the circumstances in which that 
action is taken, or do prior actions that led to the context 
in which the action is taken need to be considered?” In 
other words, how much of a student’s activities do we 
analyze? 

Additionally, the domain needs to be understood. 
Beyond a definition of technology, Pearson points out, 
we need to know more about “how people actually learn 
technological concepts.” He explains. 



Assessment designers would benefit from having 
more information about such issues as expert/ 
novice differences in knowledge acquisition, 
concept formation, misconceptions, and knowl- 
edge transfer, just to name some of the most 
obvious areas. A better understanding of these 
very fundamentalaspects of learning will greatly 
improve the quality of assessment for technologi- 
cal literacy, as well as the potential to harness 
simulation effectively. 

Panelists also identified issues beyond the functional- 
ity of the test format. For Pearson, technological literacy 
includes the ability to weigh the risks and benefits of 
new technologies, and he applies that to simulation-based 
assessment: 

I would like to ask my colleagues whether they 
feel students and teachers should, in addition to 
simply using ICT as a learning tool, also be 
challenged to think about and discuss ICT in a 
somewhat more critical way. For example, what 
are some of the unintended consequences of our 
use of computers in education?What trade-offs are 
we making, intentionally or not, by placing ICT in 
such a prominent position in U.S. education? 

In response, Ripley shifts the critical eye from 
technological priorities to assessments. He writes. 

The difficulty and challenge in focusing our 
technology ambitions on assessment is that 
assessment is such a blunt instrument. National 
assessments are filled with unintended conse- 
quences. They atrophy and become predictable 
within a matter of a couple of years, with the 
result that creativity and innovation are lost. And 
the processes of preparing students for high stakes 
national assessments are rarely witnesses of 
exemplary pedagogy. 



Edvantia is a nonprofit education research and 
development corporation, founded in 1966, that partners 
with practitioners, education agencies, publishers, and 
service providers to improve learning and advance 
student success. Edvantia provides clients with a range 
of services, including research, evaluation, professional 
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Educational Laboratory, Inc. (AEL); on September 1, 
2005, AEL became Edvantia, Inc. The Regional Educa- 
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the Appalachia Educational Laboratory at Edvantia. 
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This document summarizes an e-mail-based panel 
discussion that took place In 2005. It Is part of the Soapbox 
online forum sponsored by the Institute for the Advance- 
ment of Emerging Technologies In Education (lAETE) at 
the Appalachia Educational Laboratory. The Soapbox 
forums sponsor discussion among educators, education 
researchers, technology Industry leaders, and others 
interested in technology’s role In advancing education. To 
find complete quotes In context, access the full-text 
discussion on the Web at www.iaete.org/soapbox. 

This document is produced with funds from the 
Institute of Education Sciences (lES), U.S. Department of 
Education, under contract number ED-01-CO-0016. Its 
contents do not necessarily reflect the views of lES, the 
Department, or any other agency of the U.S. government . 
Edvantia is an equal employment opportunity/affirmative 
action employer. 
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